[YouTube] Switch to new search suggestion domain and improve error handling#1056
Conversation
…ndling - Switch to the new domain used by YouTube for search suggestions, suggestqueries-clients6.youtube.com, and add the xhr query parameter with the t value, to allow getting responses without requiring trim; - Use the Java 8 Stream API to collect search suggestions and improve invalid response detection by checking whether the content type of the response returned is JSON; - Move the licence header at the top of the file.
- Remove useless concatenation on the downloader path; - Remove unneeded public test modifier; - Update license header; - Specify the service class tested instead of the generic class.
|
Would it make sense to fall back to the old search API if the new one fails? I've observed the new API being rate limited on Piped at times, so I think it could help. |
|
Isn't the same rate limit applied when you switch the search suggestion host? It happens for YouTube manifests. |
|
I've never really checked that, will see when I get rate limited again. |
Stypox
left a comment
There was a problem hiding this comment.
Code looks good, @FireMasterK feel free to merge when you think it's ok :-)
|
|
|
Ok, so what should we do? Pick a search suggestion host randomly? |
|
We could do that indeed. I was thinking we could use a static variable to store the last time we were rate limited as a We get a 403 status code if we're rate limited. |
|
It looks like I was using ipv6 on my testing, but my backends run on ipv4 only, the rate limit is also indeed on the new API, so I think this is ready to merge! |
I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add thexhrquery parameter with thetvalue, to allow getting responses without requiring trim;Use the Java 8 Stream API to collect search suggestions and improve invalid response detection by checking whether the content type of the response returned is JSON;
Improve
YoutubeSuggestionExtractorTestand update its corresponding mocks.